{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Transfer Learning\n",
    "\n",
    "This notebook shows how to use pre-trained models from [TensorFlowHub](https://www.tensorflow.org/hub). Sometimes, there is not enough data, computational resources, or time to train a model from scratch to solve a particular problem. We'll use a pre-trained model to classify flowers with better accuracy than a new model for use in a mobile application.\n",
    "\n",
    "## Learning Objectives\n",
    "1. Know how to apply image augmentation\n",
    "2. Know how to download and use a TensorFlow Hub module as a layer in Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pathlib\n",
    "from PIL import Image\n",
    "\n",
    "import IPython.display as display\n",
    "import matplotlib.pylab as plt\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.layers import (\n",
    "    Conv2D, Dense, Dropout, Flatten, MaxPooling2D, Softmax)\n",
    "import tensorflow_hub as hub"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the data\n",
    "\n",
    "As usual, let's take a look at the data before we start building our model. We'll be using a creative-commons licensed flower photo dataset of 3670 images falling into 5 categories: 'daisy', 'roses', 'dandelion', 'sunflowers', and 'tulips'.\n",
    "\n",
    "The below [tf.keras.utils.get_file](https://www.tensorflow.org/api_docs/python/tf/keras/utils/get_file) command downloads a dataset to the local Keras cache. To see the files through a terminal, copy the output of the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = tf.keras.utils.get_file(\n",
    "    'flower_photos',\n",
    "    'https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz',\n",
    "    untar=True)\n",
    "\n",
    "# Print data path\n",
    "print(\"cd\", data_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use python's built in [pathlib](https://docs.python.org/3/library/pathlib.html) tool to get a sense of this unstructured data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = pathlib.Path(data_dir)\n",
    "\n",
    "image_count = len(list(data_dir.glob('*/*.jpg')))\n",
    "print(\"There are\", image_count, \"images.\")\n",
    "\n",
    "CLASS_NAMES = np.array(\n",
    "    [item.name for item in data_dir.glob('*') if item.name != \"LICENSE.txt\"])\n",
    "print(\"These are the available classes:\", CLASS_NAMES)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's display the images so we can see what our model will be trying to learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "roses = list(data_dir.glob('roses/*'))\n",
    "\n",
    "for image_path in roses[:3]:\n",
    "    display.display(Image.open(str(image_path)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the dataset\n",
    "\n",
    "Keras has some convenient methods to read in image data. For instance [tf.keras.preprocessing.image.ImageDataGenerator](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator) is great for small local datasets. A tutorial on how to use it can be found [here](https://www.tensorflow.org/tutorials/load_data/images), but what if we have so many images, it doesn't fit on a local machine? We can use [tf.data.datasets](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) to build a generator based on files in a Google Cloud Storage Bucket.\n",
    "\n",
    "We have already prepared these images to be stored on the cloud in `gs://cloud-ml-data/img/flower_photos/`. The images are randomly split into a training set with 90% data and an iterable with 10% data listed in CSV files:\n",
    "\n",
    "Training set: [train_set.csv](https://storage.cloud.google.com/cloud-ml-data/img/flower_photos/train_set.csv)  \n",
    "Evaluation set: [eval_set.csv](https://storage.cloud.google.com/cloud-ml-data/img/flower_photos/eval_set.csv)  \n",
    "\n",
    "Explore the format and contents of the train.csv by running:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gcloud storage cat gs://cloud-ml-data/img/flower_photos/train_set.csv | head -5 > /tmp/input.csv\n",    "!cat /tmp/input.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gcloud storage cat gs://cloud-ml-data/img/flower_photos/train_set.csv  | sed 's/,/ /g' | awk '{print $2}' | sort | uniq > /tmp/labels.txt\n",
    "!cat /tmp/labels.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's figure out how to read one of these images from the cloud. TensorFlow's [tf.io.read_file](https://www.tensorflow.org/api_docs/python/tf/io/read_file) can help us read the file contents, but the result will be a [Base64 image string](https://en.wikipedia.org/wiki/Base64). Hmm... not very readable for humans or Tensorflow.\n",
    "\n",
    "Thankfully, TensorFlow's [tf.image.decode_jpeg](https://www.tensorflow.org/api_docs/python/tf/io/decode_jpeg) function can decode this string into an integer array, and [tf.image.convert_image_dtype](https://www.tensorflow.org/api_docs/python/tf/image/convert_image_dtype) can cast it into a 0 - 1 range float. Finally, we'll use [tf.image.resize](https://www.tensorflow.org/api_docs/python/tf/image/resize) to force image dimensions to be consistent for our neural network.\n",
    "\n",
    "We'll wrap these into a function as we'll be calling these repeatedly. While we're at it, let's also define our constants for our neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "IMG_HEIGHT = 224\n",
    "IMG_WIDTH = 224\n",
    "IMG_CHANNELS = 3\n",
    "\n",
    "BATCH_SIZE = 32\n",
    "# 10 is a magic number tuned for local training of this dataset.\n",
    "SHUFFLE_BUFFER = 10 * BATCH_SIZE\n",
    "AUTOTUNE = tf.data.experimental.AUTOTUNE\n",
    "\n",
    "VALIDATION_IMAGES = 370\n",
    "VALIDATION_STEPS = VALIDATION_IMAGES // BATCH_SIZE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decode_img(img, reshape_dims):\n",
    "    # Convert the compressed string to a 3D uint8 tensor.\n",
    "    img = tf.image.decode_jpeg(img, channels=IMG_CHANNELS)\n",
    "    # Use `convert_image_dtype` to convert to floats in the [0,1] range.\n",
    "    img = tf.image.convert_image_dtype(img, tf.float32)\n",
    "    # Resize the image to the desired size.\n",
    "    return tf.image.resize(img, reshape_dims)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is it working? Let's see!\n",
    "\n",
    "**TODO 1.a:** Run the `decode_img` function and plot it to see a happy looking daisy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img = tf.io.read_file(\n",
    "    \"gs://cloud-ml-data/img/flower_photos/daisy/754296579_30a9ae018c_n.jpg\")\n",
    "\n",
    "# Uncomment to see the image string.\n",
    "#print(img)\n",
    "# TODO: decode image and plot it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One flower down, 3669 more of them to go. Rather than load all the photos in directly, we'll use the file paths given to us in the csv and load the images when we batch. [tf.io.decode_csv](https://www.tensorflow.org/api_docs/python/tf/io/decode_csv) reads in csv rows (or each line in a csv file), while [tf.math.equal](https://www.tensorflow.org/api_docs/python/tf/math/equal) will help us format our label such that it's a boolean array with a truth value corresponding to the class in `CLASS_NAMES`, much like the labels for the MNIST Lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decode_csv(csv_row):\n",
    "    record_defaults = [\"path\", \"flower\"]\n",
    "    filename, label_string = tf.io.decode_csv(csv_row, record_defaults)\n",
    "    image_bytes = tf.io.read_file(filename=filename)\n",
    "    label = tf.math.equal(CLASS_NAMES, label_string)\n",
    "    return image_bytes, label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll transform the images to give our network more variety to train on. There are a number of [image manipulation functions](https://www.tensorflow.org/api_docs/python/tf/image). We'll cover just a few:\n",
    "\n",
    "* [tf.image.random_crop](https://www.tensorflow.org/api_docs/python/tf/image/random_crop) - Randomly deletes the top/bottom rows and left/right columns down to the dimensions specified.\n",
    "* [tf.image.random_flip_left_right](https://www.tensorflow.org/api_docs/python/tf/image/random_flip_left_right) - Randomly flips the image horizontally\n",
    "* [tf.image.random_brightness](https://www.tensorflow.org/api_docs/python/tf/image/random_brightness) - Randomly adjusts how dark or light the image is.\n",
    "* [tf.image.random_contrast](https://www.tensorflow.org/api_docs/python/tf/image/random_contrast) - Randomly adjusts image contrast.\n",
    "\n",
    "**TODO 1.b:** Augment the image using the random functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_DELTA = 63.0 / 255.0  # Change brightness by at most 17.7%\n",
    "CONTRAST_LOWER = 0.2\n",
    "CONTRAST_UPPER = 1.8\n",
    "\n",
    "\n",
    "def read_and_preprocess(image_bytes, label, random_augment=False):\n",
    "    if random_augment:\n",
    "        img = decode_img(image_bytes, [IMG_HEIGHT + 10, IMG_WIDTH + 10])\n",
    "        # TODO: augment the image.\n",
    "    else:\n",
    "        img = decode_img(image_bytes, [IMG_WIDTH, IMG_HEIGHT])\n",
    "    return img, label\n",
    "\n",
    "\n",
    "def read_and_preprocess_with_augment(image_bytes, label):\n",
    "    return read_and_preprocess(image_bytes, label, random_augment=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we'll make a function to craft our full dataset using [tf.data.dataset](https://www.tensorflow.org/api_docs/python/tf/data/Dataset). The [tf.data.TextLineDataset](https://www.tensorflow.org/api_docs/python/tf/data/TextLineDataset) will read in each line in our train/eval csv files to our `decode_csv` function.\n",
    "\n",
    "[.cache](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#cache) is key here. It will store the dataset in memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_dataset(csv_of_filenames, batch_size, training=True):\n",
    "    dataset = tf.data.TextLineDataset(filenames=csv_of_filenames) \\\n",
    "        .map(decode_csv).cache()\n",
    "\n",
    "    if training:\n",
    "        dataset = dataset \\\n",
    "            .map(read_and_preprocess_with_augment) \\\n",
    "            .shuffle(SHUFFLE_BUFFER) \\\n",
    "            .repeat(count=None)  # Indefinately.\n",
    "    else:\n",
    "        dataset = dataset \\\n",
    "            .map(read_and_preprocess) \\\n",
    "            .repeat(count=1)  # Each photo used once.\n",
    "\n",
    "    # Prefetch prepares the next set of batches while current batch is in use.\n",
    "    return dataset.batch(batch_size=batch_size).prefetch(buffer_size=AUTOTUNE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll test it out with our training set. A batch size of one will allow us to easily look at each augmented image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_path = \"gs://cloud-ml-data/img/flower_photos/train_set.csv\"\n",
    "train_data = load_dataset(train_path, 1)\n",
    "itr = iter(train_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO 1.c:** Run the below cell repeatedly to see the results of different batches. The images have been un-normalized for human eyes. Can you tell what type of flowers they are? Is it fair for the AI to learn on?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_batch, label_batch = next(itr)\n",
    "img = image_batch[0]\n",
    "plt.imshow(img)\n",
    "print(label_batch[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** It may take a 4-5 minutes to see result of different batches. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MobileNetV2\n",
    "\n",
    "These flower photos are much larger than handwritting recognition images in MNIST. They are about 10 times as many pixels per axis **and** there are three color channels, making the information here over 200 times larger!\n",
    "\n",
    "How do our current techniques stand up? Copy your best model architecture over from the <a href=\"2_mnist_models.ipynb\">MNIST models lab</a> and see how well it does after training for 5 epochs of 50 steps.\n",
    "\n",
    "**TODO 2.a** Copy over the most accurate model from 2_mnist_models.ipynb or build a new CNN Keras model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_path = \"gs://cloud-ml-data/img/flower_photos/eval_set.csv\"\n",
    "nclasses = len(CLASS_NAMES)\n",
    "hidden_layer_1_neurons = 400\n",
    "hidden_layer_2_neurons = 100\n",
    "dropout_rate = 0.25\n",
    "num_filters_1 = 64\n",
    "kernel_size_1 = 3\n",
    "pooling_size_1 = 2\n",
    "num_filters_2 = 32\n",
    "kernel_size_2 = 3\n",
    "pooling_size_2 = 2\n",
    "\n",
    "layers = [\n",
    "    # TODO: Add your image model.\n",
    "]\n",
    "\n",
    "old_model = Sequential(layers)\n",
    "old_model.compile(\n",
    "    optimizer='adam',\n",
    "    loss='categorical_crossentropy',\n",
    "    metrics=['accuracy'])\n",
    "\n",
    "train_ds = load_dataset(train_path, BATCH_SIZE)\n",
    "eval_ds = load_dataset(eval_path, BATCH_SIZE, training=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "old_model.fit_generator(\n",
    "    train_ds,\n",
    "    epochs=5,\n",
    "    steps_per_epoch=5,\n",
    "    validation_data=eval_ds,\n",
    "    validation_steps=VALIDATION_STEPS\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If your model is like mine, it learns a little bit, slightly better then random, but *ugh*, it's too slow! With a batch size of 32, 5 epochs of 5 steps is only getting through about a quarter of our images. Not to mention, this is a much larger problem then MNIST, so wouldn't we need a larger model? But how big do we need to make it?\n",
    "\n",
    "Enter Transfer Learning. Why not take advantage of someone else's hard work? We can take the layers of a model that's been trained on a similar problem to ours and splice it into our own model.\n",
    "\n",
    "[Tensorflow Hub](https://tfhub.dev/s?module-type=image-augmentation,image-classification,image-others,image-style-transfer,image-rnn-agent) is a database of models, many of which can be used for Transfer Learning. We'll use a model called [MobileNet](https://tfhub.dev/google/imagenet/mobilenet_v2_035_224/feature_vector/4) which is an architecture optimized for image classification on mobile devices, which can be done with [TensorFlow Lite](https://github.com/tensorflow/hub/blob/master/examples/colab/tf2_image_retraining.ipynb). Let's compare how a model trained on [ImageNet](http://www.image-net.org/) data compares to one built from scratch.\n",
    "\n",
    "The `tensorflow_hub` python package has a function to include a Hub model as a [layer in Keras](https://www.tensorflow.org/hub/api_docs/python/hub/KerasLayer). We'll set the weights of this model as un-trainable. Even though this is a compressed version of full scale image classification models, it still has over four hundred thousand paramaters! Training all these would not only add to our computation, but it is also prone to over-fitting. We'll add some L2 regularization and Dropout to prevent that from happening to our trainable weights.\n",
    "\n",
    "**TODO 2.b**: Add a Hub Keras Layer at the top of the model using the handle provided."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "module_selection = \"mobilenet_v2_100_224\"\n",
    "module_handle = \"https://tfhub.dev/google/imagenet/{}/feature_vector/4\" \\\n",
    "    .format(module_selection)\n",
    "\n",
    "transfer_model = tf.keras.Sequential([\n",
    "    # TODO\n",
    "    tf.keras.layers.Dropout(rate=0.2),\n",
    "    tf.keras.layers.Dense(\n",
    "        nclasses,\n",
    "        activation='softmax',\n",
    "        kernel_regularizer=tf.keras.regularizers.l2(0.0001))\n",
    "])\n",
    "transfer_model.build((None,)+(IMG_HEIGHT, IMG_WIDTH, IMG_CHANNELS))\n",
    "transfer_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even though we're only adding one more `Dense` layer in order to get the probabilities for each of the 5 flower types, we end up with over six thousand parameters to train ourselves. Wow!\n",
    "\n",
    "Moment of truth. Let's compile this new model and see how it compares to our MNIST architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "transfer_model.compile(\n",
    "    optimizer='adam',\n",
    "    loss='categorical_crossentropy',\n",
    "    metrics=['accuracy'])\n",
    "\n",
    "train_ds = load_dataset(train_path, BATCH_SIZE)\n",
    "eval_ds = load_dataset(eval_path, BATCH_SIZE, training=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "transfer_model.fit(\n",
    "    train_ds,\n",
    "    epochs=5,\n",
    "    steps_per_epoch=5,\n",
    "    validation_data=eval_ds,\n",
    "    validation_steps=VALIDATION_STEPS\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright, looking better!\n",
    "\n",
    "Still, there's clear room to improve. Data bottlenecks are especially prevalent with image data due to the size of the image files. There's much to consider such as the computation of augmenting images and the bandwidth to transfer images between machines.\n",
    "\n",
    "Think life is too short, and there has to be a better way? In the next lab, we'll blast away these problems by developing a cloud strategy to train with TPUs!\n",
    "\n",
    "## Bonus Exercise\n",
    "\n",
    "Keras has a [local way](https://keras.io/models/sequential/) to do distributed training, but we'll be using a different technique in the next lab. Want to give the local way a try? Check out this excellent [blog post](https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly) to get started. Or want to go full-blown Keras? It also has a number of [pre-trained models](https://keras.io/applications/) ready to use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2019 Google Inc.\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at\n",
    "http://www.apache.org/licenses/LICENSE-2.0\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
